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1 IMMEDIATE VERIFICATION OF PRINTED COPY 

2 FIELD OF THE INVENTION 

3 This application relates to the field of high-speed digital 

4 printing. It is more specifically concerned with the immediate 

5 automatic verification of printed copy to determine the intended 

6 presence of ink on paper and the absence of unintended ink or 

7 other marks on paper. 
8 

9 BACKGROUND OF THE INVENTION 

10 Definition of terms and constructs 

11 Prior to the application of ink to paper which creates a printed 

12 copy, a digitized image of the matter to be printed is created. 

13 As used herein, a digitized image is an abstraction of a physical 

14 image that has been created or scanned. It is stored in a 

15 computer's memory as rectangular arrays of numbers corresponding 

16 to that image's (one or more) color planes. Each array element 

17 corresponds to a very small area of the printed image and is 

18 called a picture element, or pixel. The numeric value associated 
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1 with each pixel for a monochrome image represents the magnitude 

2 of the average brightness of its single color (for example, 

3 black) plane. 

4 If the digitized image has been converted from continuous tone 

5 picture elements to halftone picture elements, the halftone 

6 picture elements will be referred to herein as pels and their 

7 color component values referred to as ink-density values. As with 

8 pixels, a different value is associated with each different one 

9 of the image's color planes for each pel, and the number of color 

10 planes in the halftone representation may be greater than the 

11 number of color planes in the digitized image. Thus, the 

12 digitized image and the copy printed from the halftone image (the 

13 printed copy) are two distinct, but related, representations of 

14 the same physical image. 

15 Herein the word halftone will be taken to mean that gradations 

16 from light to dark in pels that are obtained by the relative 

17 darkness and density of tiny dots of inks that are to be applied 

18 to paper or other substrate material. Also, if the digitized 

19 image is a color image, its pixel values are ordinarily the 

20 relative brightness values of additive radiant primary colors, 

21 such as those of a computer's display. Therefore, the halftone 

22 conversion process as referred to herein also includes conversion 

23 of the pixel values of radiant primary colors into the pel values 



YOR9-2003-0344 



-2- 



1 of light absorbing primary colors (for example, Cyan, Magenta, 

2 Yellow and Black ink densities) that are needed for printing. The 

3 halftone image then may be printed on paper or other substrate 

4 material; such printed image is herein called a printed copy. 

5 Whenever reference is made herein to color planes, it is 

6 understood to include any number of color planes used by a 

7 particular image's digitizing technique to define the pixel's or 

8 pel's color characteristics. Pixel values, as well as pel values, 

9 have a magnitude represented by at least one binary digit or bit. 

10 Whenever reference is made herein to ink or ink-density value, it 

11 is understood to refer to any substance that is used to apply 

12 color to paper or other substrate material, be that substance 

13 ink, dye, toner or other. Further, ink-density values range from 

14 0% to 100%, meaning from no ink applied to the area of a picture 

15 element [pel] on paper up to total coverage of that area on 

16 paper. 

17 A high-speed digital printer accepts a first stream of digitized 

18 images, called source images, each source image representing a 

19 specific page to be printed, and converts those digitized images 

20 into printed copies by depositing dots of black or colored inks 

21 onto paper. The spatial density and positions of the dots of ink 

22 on the printed copies are directly related to and defined by the 
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1 numerical values of the digitized image pixels. The source of 

2 paper or other substrate material used in high-speed printers may 

3 be either cut sheets or a continuous seamless web of paper 

4 unrolled as needed from a large roll. 

5 To effect comparison of a digitized source image (which is herein 

6 stipulated to be a perfect representation of what is intended to 

7 be printed) with a scanned image created by scanning a printed 

8 copy of the source image requires common features within the two 

9 images to be positionally aligned. Prior art exists for aligning 

10 and comparing two images. In the prior art, the two images are 

11 positionally aligned either by semi-automated or fully automated 

12 methods. The alignment methods uses distinct common features 

13 within both images to effect positional alignment of all, or 

14 segmented parts of, the two images. Three examples of prior art 

15 are: 

16 1. Braudaway, G. , * Recovering Invisible Image Watermarks from 

17 Images Distorted by the x Stirmark' Algorithm," Proceedings 

18 of the IS&T PICS Conference, April 25-28, 1999, pp. 307-310; 

19 2. Braudaway, G. And Mintzer, F . , 11 Automatic recovery of 

20 invisible image watermarks from geometrically distorted 

21 images," Journal of Electronic Imaging, October 2000, Volume 

22 9(4) , pp. 477-483; and 

23 3. "RECOVERING AN INVISIBLE DIGITAL IMAGE FROM A DISTORTED 



YOR9-2003-0344 



-4- 



1 IMAGE REPLICA," U.S. Patent 6571021, issued May 23, 2003. 

2 These three examples of prior art are herein incorporated by 

3 reference in their entirety and for all purposes. 

4 PROBLEMS WITH THE PRIOR ART 

5 A first category of printing defects, called significant defects, 

6 are defects that could cause misinterpretation of a single 

7 printed character by a human reader. A significant defect is 

8 herein defined to be one of two types, 1) either a small square 

9 area having black pel values in the first-stream image that are 

10 not found in corresponding pel values of the printed copy, or 2) 

11 ink or other marking detected in pel values of the printed copy 

12 where corresponding pel values of the first-stream are white. 

13 Further, to be a significant defect, that is, to potentially 

14 cause misinterpretation by a human reader, the unexpected small 

15 square of either type must be in the near proximity of a 

16 character of text that is intended to be printed. An example 

17 small square is 0.01 inches on a side, and an example near 

18 proximity is 0.04 inches in any direction. Either of the two 

19 types of significant defects constitute a printing error, and the 

20 page on which it is detected is recorded in an error log of 

21 defective pages . 



22 A second category of printing defects, called cosmetic defects, 
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1 are defects that would probably not cause misinterpretation of a 

2 single printed character by a human reader but are still 

3 objectionable in the printed copy. Cosmetic defects include 

4 stains on the paper, unintended printed streaks and the like. 

5 Cosmetic defects, based on their size and frequency of 

6 occurrence, may constitute printing errors, and if judged so, the 

7 pages on which they are detected are also recorded in an error 

8 log of defective pages . 

9 An example anticipated application of the cited prior art is to 

10 the verification of a mortgage loan commitment letter sent by a 

11 bank to a mortgagee for signature. The bank creates the letter, 

12 containing an identifying loan application number, as a printed 

13 copy of a digitized source image, sends the letter to the 

14 mortgagee, and retains the digitized source image for later 

15 verification of the returned letter. The returned letter is 

16 matched to the retained source image using the embedded 

17 identifying loan application number (usually by human inspection) 

18 and is scanned to produce a scanned image. The scanned image is 

19 automatically aligned with the source image using three or more 

20 corresponding common features of the two images, according to the 

21 methods taught in the prior art of References 2 and 3 . 

22 A single composite color image is formed from the two digitized 

23 images, the source image and the aligned scanned image, with the 
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1 red color plane made exclusively from pixels of the source image 

2 and the blue and green planes made identically and exclusively 

3 from pixels of the aligned scanned image. The composite color 

4 image is displayed on a CRT display for human observation. If the 

5 scanned image is aligned perfectly with the source image, a 

6 monochrome black-white image will result. If alignment is 

7 imperfect, cyan and red fringes will appear around the black text 

8 on the white surround in the composite image. More importantly, 

9 when displayed to a human viewer, gross differences, such as an 

10 expected signature or unexpected additions, modifications or 

11 deletions of text will show as conspicuous red or cyan features 

12 in the composite image, and a human decision can be made to 

13 accept the signed letter or to forward an obviously modified 

14 letter to the bank's fraud control department for further 

15 analysis. 

16 Significant unsolved problems become evident when attempting to 

17 adapt the cited prior art to high speed digital printing. 

18 Firstly, positional alignment of common features found within two 

19 images requires selection of a source image that corresponds to 

20 the candidate scanned image to be accomplished by some 

21 independent means. The most commonly used independent means 

22 requires a human being to view and recognize a common embedded 

23 feature, such as a loan application number, the inclusion of 

24 which may be unallowable in other applications. Secondly, the 
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1 need to automatically match a stream of source images with many 

2 sequentially printed copies of each of those source images, and 

3 simultaneously, to identify the place in the sequence where each 

4 defective printed copy lies has not been anticipated. Thirdly, 

5 although adequate for the purposes stated in the prior art and 

6 for the example verification of a mortgage commitment letter, the 

7 accuracy of alignment achievable using the prior art is 

8 insufficient to meet the requirements for automated high 

9 precision printing verification. As in the example given, the 

10 feature size allowable for undetected defects must be smaller 

11 than a square having 0.01 inch sides or have feature proximity 

12 greater than 0.04 inches in any direction from a readable 

13 character of text. For printed copies on paper, these tolerances 

14 can not be met using methods taught in the prior art. To 

15 compensate for the poor dimensional stability of paper, meeting 

16 the tolerances requires page by page, line by line, and pixel 

17 (pel) by pixel (pel) alignment beyond their capabilities. Thus, 

18 the required accuracy creates a need for an additional and 

19 unanticipated method of fine alignment. 

20 ASPECTS OF THE INVENTION 

21 An aspect of the present invention is an improved system and 

22 method for detecting defects in a printed copy. 
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1 An aspect of the present invention is an improved system and 

2 method for explicit detection and logging of significant and 

3 cosmetic defects in a printed copy. 

4 SUMMARY OF THE INVENTION 

5 A first stream of digitized images (sources images) is received 

6 by a computer system. The source stream is printed to create one 

7 or more printed copies. Print verification is done by scanning 

8 the printed copies, thereby forming a second stream of digitized 

9 images of the specific pages. Digitized images from the second 

10 stream are then spatially aligned in a preferred embodiment page 

11 by page, line by line and pixel (pel) by pixel (pel) with 

12 corresponding digitized images from the first stream. Once pairs 

13 of corresponding pages are aligned, one image from the first 

14 stream, called a first-stream image, and a corresponding image 

15 from the second stream, called a second-stream image, the pixel 

16 values of the second-stream image are, in a preferred embodiment, 

17 converted to pel values that have same number of bits (e.g., 

18 one-bit) as the corresponding pel values in the source image. The 

19 aligned image lines of both the first-stream and second-stream 

20 images, now both having a same number pel values (e.g., one-bit) 

21 per pel, are compared to find pel sequences that are different. 

22 These differences represent defects in the printed page. 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 



2 These and other aspects, features, and advantages of the present 

3 invention will become apparent upon further consideration of the 

4 following detailed description of the invention when read in 

5 conjunction with the drawing figures, in which: 

6 Figure 1 shows a functional block diagram of an example 

7 high-speed printer train using the present invention. 

8 Figure 1A is a block diagram of one preferred embodiment of the 

9 Print Verification Unit of the present system. 
10 

11 Figure 2 shows the format and placement of 

12 synchronization-strips, in one preferred embodiment, that are 

13 included in all source-images to be printed on all web segments - 

14 A section of the synchronization-strip is enlarged to show its 

15 detail. 

16 Figure 3 shows a perforated web segment as it would appear had it 

17 been printed from by an ideal printer on ideal paper. 

18 Figure 4 shows a perforated web segment as it would appear had it 

19 been printed by a typical printer on typical paper, with the 
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1 resulting distortion exaggerated for illustrative purposes. 

2 Figure 5 shows defining tt truth- tables" for the logical or, and 

3 and exclusive-or operators. 

4 Figure 6 shows typical horizontal paper distortion measured by 

5 methods of the present invention. 

6 Figure 7 shows a normal image along with dilation and erosion 

7 masks created from the normal image using methods of the present 

8 invention. 

9 Figure 8 is a flow chart of a process for course image alignment. 

10 Figure 9 is a flow chart of a sub-process for fine image 

11 alignment. 

12 Figure 10 is a flow chart of a sub-process for comparison of a 

13 printed copy with a source image. 

14 DETAILED DESCRIPTION OF THE INVENTION 

15 The invention described herein specifies improved techniques for 

16 verifying the quality of a printed copy. Herein, verifying the 

17 quality of a printed copy is taken to mean determining the 
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1 presence of ink intended to have been applied to the printed 

2 copy, and determining the absence of unintended coloration, be it 

3 from ink or other source, on the printed copy. 

4 The preferred embodiment of the present invention will use a 

5 roll-fed paper web. Those skilled in the art will understand that 

6 the concepts of the present invention are easily adaptable to 

7 sheet fed printers as well, and to substrate materials other than 

8 paper . 

9 An example high-speed printer is shown in Figure 1. Referring to 

10 Figure 1, the units (101) through (117) are collectively referred 

11 to as the printer or print-train. The units of the print- train, 

12 although physically separate, are threaded with a common 

13 continuous strip of paper, called a web. One of the physically 

14 separate units is designated as the Master in terms of 

15 establishing the speed of the web, and all other units in the 

16 train independently synchronize their speeds with that of the 

17 Master by sensing tension in the paper web. 

18 In the example, well known print- train, the web is presented as 

19 roll of paper (101) nineteen inches wide and 40,000 to 50,000 

20 feet in length, depending on paper thickness. Commonly available 

21 paper rolls are fifty inches in diameter with a six inch core. 

22 The paper roll is mounted into a servomotor controlled Unwinder 
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1 unit (103) that unwinds and supplies the web to the print train 

2 at a web speed determined. by a Master unit. In the example 

3 print- train, the Obverse Print Engine (107) is designated as the 

4 Master for web speed purposes. In a preferred embodiment, the 

5 continuous web is perforated into eleven inch web- segments , and 

6 guide holes are punched into tear-strips at each edge of the web 

7 by the Perforator /Hole Punch unit (105) . Other web-segments are 

8 envisioned (e.g., for metric standards, drawing formats, etc.) 

9 Two 8 %" by 11" page impressions , each defined by a source image, 

10 are printed on the obverse side of each web segment by the 

11 Obverse Print Engine (107) . The web is physically inverted by the 

12 Web Inverter unit (109) and passes into the Reverse Print Engine 

13 (111) . Two additional page impressions are printed on the reverse 

14 side of the web by the Reverse Print Engine. The four impressions 

15 printed on each web segment are logically oriented and sequenced 

16 by the Printer Controller (119) before being sent to the Obverse 

17 and Reverse Print Engines so pages in a final multi-page document 

18 will be in the correct order. 

19 The web then passes through a novel Print Verification Unit (113) 

20 where both the obverse and reverse sides of the web are scanned 

21 to produce two streams of digitized images, each called 

22 second-stream images, one second stream image from the top 

23 (reverse) and one second stream image from the bottom (obverse) 

24 of the web. These second-stream images are to be aligned with and 
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1 compared to corresponding first-stream images to find small areas 

2 that are different, called defects. 

3 The web continues on into the well known Accumulator unit (115) 

4 and Fan Folder unit (117) . The Accumulator unit is a mechanical 

5 web buffer that can accumulate a significant length of the web 

6 from its continuous input with no output before it must signal a 

7 web stop. The Accumulator unit facilitates brief intermittent 

8 downstream web stopping by the Fan Folder unit, as it separates 

9 and delivers a fan- folded document or short sequence of documents 

10 onto a conveyor, and then at a variable speed resumes folding the 

11 pages of the next document on the web. The documents are 

12 fan- folded and separated using the web perforations created by 

13 the Perforator /Hole Punch unit (105) . The example print- train can 

14 produce in excess of 200 page impressions (50 perforated web 

15 segments) per minute. 

16 Figure la shows a block diagram of one preferred embodiment of 

17 the Print Verification Unit. The paper web (150) is threaded 

18 through the unit and passes over two curved platens, (151) and 

19 (152). The web is illuminated by two sources (154) and (155) as 

20 it passes over the two platens. Tension in the wed is controlled 

21 to keep the web in physical contact with the platens so the two 

22 linear image scanning arrays (157) and (158) can maintain a fixed 

23 focus on the web surface. Image scanning array (157) scans and 
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1 samples the obverse side of the web to produces sequential lines 

2 of pixels that are grouped into scanned images, and image 

3 scanning array (158) does the same for the reverse side of the 

4 web. 

5 The Print Verification Unit contains a digital computer composed 

6 of a Central Processing Unit (160), an associated memory (161), 

7 and an input/output interface (162) used for communication with 

8 the Printer Controller [Figure 1, (119)]. A first stream of 

9 digitized source images are received via the input/output 

10 interface (162) and stored in the computer memory (165). The 

11 scanned image lines produced by the two linear image scanning 

12 arrays (157) and (158) are organized into two streams of 

13 digitized scanned images and are also stored in the computer 

14 memory (166) . Computer processes that perform source image to 

15 scanned image alignment (168) and processes that perform page by 

16 page, line by line and pel by pel comparison of corresponding 

17 source and scanned images (169) are also stored in the computer 

18 memory. Defects found in the printed copies (determined by 

19 processing the streams of scanned images) are reported to the 

20 Printer Controller via the input/output interface (162) . 

21 Coarse alignment of images of the first and second streams 

22 In a preferred embodiment, spatial alignment of digitized images 
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1 from the second stream with those of the first stream is 

2 accomplished in two phases, a coarse alignment phase and fine 

3 alignment phase. The objective of coarse alignment is to locate 

4 lines in the second stream that correspond to lines in the first 

5 stream. In a preferred embodiment, this is facilitated by a 

6 linear reference markers, called synchronization-strips , that are 

7 added to all digitized source images. In the preferred 

8 embodiment, synchronization-strips are positioned to be printed 

9 in the left and right sacrificial margins and the center gutter 

10 [between page impressions] of the web. The margins are 

11 sacrificial in that they are trimmed off after the final document 

12 is bound, and the center gutter of the web is not visible after 

13 binding unless the bound document is disassembled. Use of 

14 synchronization-strips is one of the more significant aspects of 

15 the present invention. 

16 Referring to Figure 2, the outlined rectangle (201) represents 

17 the edges of a printed-copy [one page impression] . Lines of the 

18 printed-copy lie horizontally across the printed-copy as shown 

19 and also across the paper web. In the example embodiment, two 

20 printed-copies, each a document page, lie side-by-side on each 

21 side of each web segment. An example synchronization-strip, 

22 aligned vertically and along the web is printed at the trim edge 

23 (203) and at the bound edge (205) of the printed-copy. Figure 2 

24 shows only the right half of a web segment; a second printed-copy 
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1 lies to the left, and the two printed-copies share a common 

2 center gutter synchronization-strip (205) . An enlargement of the 

3 synchronization-strip shows a basic pattern, delineated by a 

4 brace (207), that is repeated along the edges of the source image 

5 every 100 lines. At defined intervals in the 

6 synchronization-strip the basic pattern is replaced by a counter 

7 pattern, delineated by a second brace (209). In the present 

8 example, the counter pattern represents a ten-bit binary counter. 

9 Three counter patterns are placed in the synchronization-strips 

10 along the web edges and center gutter of every web segment. 

11 Concatenating bits of the three counter patterns yields a 

12 thirty-bit binary counter. The filled square in the counter 

13 pattern lying rightmost (211) represents a zero for that 

14 particular bit in the counter and a filled square in the counter 

15 pattern lying leftmost (213) represents a one. Thus, if the 

16 enlarged counter pattern is the second of three patterns, and if 

17 the filled squares in the first and third patterns are all zeros, 

18 the 3 0-bit binary number represented by the three concatenated 

19 segments is: 

20 0000000000,0000000010,0000000000 or 2 11 , 

21 which, as a decimal number is 2048. With the 30-bit binary 

22 number, a range of equivalent decimal numbers from 0 to 

23 1,073,741,823 can be represented. The thirty bit counter, 
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1 incremented for every first-stream digitized image, can thus 

2 serve as a unique page counter for a large number of documents 

3 having a large number of pages before it recycles. 

4 The synchronization-strips that are embedded into the 

5 first-stream digitized images will, when printed, exist in 

6 printed copies, and when the printed copies are scanned by the 

7 Print Verification Unit (Figure 1, (113) ) , the 

8 synchronization-strips will also exist in the second-stream 

9 digitized images. Coarse alignment of corresponding images of the 

10 first stream and second stream relies on matching the patterns in 

11 the synchronization-strips to within a fraction of a line width 

12 and a fraction of a pel position. 

13 Images in the second stream, produced by scanning the printed web 

14 segments, often have fewer lines per inch than do images in the 

15 first stream. For example, digitized images in the first stream 

16 may have 600 lines per inch while scanned images in the second 

17 stream may have only 2 50 lines per inch. Thus, to be able to 

18 match images of the two streams line by line, additional lines of 

19 the second image must be created between the available scanned 

20 lines using interpolation methods. Conversely, images of the 

21 first stream could be reduced to have 250 lines per inch using 

22 any of many image reduction methods known to those skilled in the 

23 art. The preferred embodiment, however, is to enlarge images in 
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1 the second stream by interpolation, when needed. 

2 To facilitate interpolation between the scanned lines of images 

3 in the second stream, each scanned line of every web segment is 

4 tagged with three tag-pairs, each determined by the independent 

5 tracking of the left, center and right synchronization-strip. The 

6 term tracking as used herein refers to the numeric evaluation of 

7 horizontal and vertical coordinates of sequential and 

8 specifically identifiable features in the synchronization-strips 

9 embedded in the digitized scanned images, and relating those 

10 values to known numeric values of horizontal and vertical 

11 coordinates of the same sequential and specifically identifiable 

12 features embedded in the corresponding source image. An example 

13 embodiment of a tracker uses localized application of a 

14 two-dimensional cross-correlation function, repeatedly and 

15 sequentially applied, to determine the values of horizontal and 

16 vertical coordinates of the features in the digitized scanned 

17 images, with linear interpolation of the coordinate values 

18 between applications. Those skilled in the art will recognize 

19 that the tracking method can be embodied in a number of 

20 functionally equivalent ways that produce the same or equivalent 

21 results. 

22 The second number of each of the tag-pairs is the closest 

23 matching line number of a line in a corresponding digitized image 
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1 in the first stream. If the scanning element of the Print 

2 Verification Unit (115) is perfectly aligned at right angle to 

3 the web, and if the paper is not physically warped or skewed, 

4 these three numbers will be the same. However, in most instances 

5 the three numbers will not be the same because of scanning 

6 element misalignment and paper warpage and skew. For the example 

7 pixel and pel resolutions, line 250 of a scanned image could have 

8 three tag pairs with the different second numbers, such as 

9 {...,600.28}, {...,598.85}, and {...,599.65}. 

10 Again using the previous example, digitized images in the first 

11 stream have 600 pels per inch within each line, while digitized 

12 images in the second stream have only 250 pixels per inch within 

13 each line. To be able to closely match individual pel to pel 

14 positions within corresponding pairs of images in the two 

15 streams, additional pixels in the lines of the second image must 

16 also be created using interpolation. Thus the interpolation 

17 method needed is a two-dimensional method capable of 

18 interpolating between available lines to produce intermediate 

19 lines and interpolating between available pixels in those lines 

20 to produce intermediate pixel values. 

21 As an aid to interpolation between pixel values, each 

22 synchronization-strip contains a lone vertical mark (215) . The 

23 pixel positions of the three vertical marks in lines of the 



YOR9-2003-0344 



-20- 



1 scanned images of the second stream are also tracked, and the 

2 pixel position of each of the three is recorded as the first 

3 number in its appropriate tag-pair. The first and second number 

4 in a tag-pair are, therefore, 1) the horizontal coordinate (pixel 

5 position across the web) of the appropriate lone vertical mark 

6 within the scanned line, and 2) the closest matching line number 

7 of a line in a corresponding source image in the first stream, 

8 each specified to a fraction of a pixel or fraction of a line. 

9 The three tag-pairs for each scanned line will be denoted as 

10 P{u\y x ), P{u\y 2 ) and P<Mi,yl). 

11 As an example, for a 19 inch web and source images having 600 

12 pels per inch, pel positions of the three lone vertical marks in 

13 the three synchronization-strips, denoted as Y\ , 72, and Y3 , 

14 could be positions 70, 5730 and 11390, and by design would remain 

15 at those specified pel positions for every web-segment. The three 

16 tracked lone vertical marks in the scanned image at the 

17 particular scanned line 250 might be 39.17, 2392.50 and 4755.83. 

18 The particular scanned line number is designated as L. The three 

19 complete tag-pairs for Z,=250 of the scanned images would 

20 therefore be {39.17, 600.28}, {2392.50, 598.85}, and {4755.83, 

21 599.65}. The line number of scanned image lines, the three 

22 tag-pairs for each of those scanned lines, and the specified 

23 locations of the three lone vertical marks in the three 

24 synchronization-strips of the source images are the necessary 
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1 prerequisites for the required two-dimensional (horizontal and 

2 vertical) pixel interpolation. 

3 Referring to Figure 3 and Figure 4, and for the example print 

4 train, web segments are 19 inches wide and 11 inches high. Figure 

5 3 shows a perforated web segment as it would appear if it could 

6 be printed as specified by an ideal printer on ideal paper; it 

7 would be a physical embodiment of the digitized source images and 

8 synchronization-strips without defect. The three 

9 synchronization-strips (301), (303), and (305) are shown as they 

10 would appear in the vertical direction, parallel to edges of the 

11 web. Figure 4 shows a perforated web segment as it would appear 

12 when printed by a typical printer on typical paper and 

13 redigitized by scanning to form a scanned image. The resulting 

14 spatial distortions are exaggerated for illustrative purposes. 

15 The three corresponding synchronization-strips are (401), (403), 

16 and (405) . 

17 In the preferred embodiment, six coordinate-pairs from each image 

18 in the first stream and six coordinate-pairs from a corresponding 

19 image in the second stream are used for coarse alignment of a 

20 strip of the web. The six coordinate-pairs chosen are numbered 

21 sequentially (307) through (312) for the source image and (407) 

22 through (412) for the corresponding scanned image. An example 

23 height of the strip is one inch, or 600 lines of the source 
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1 image. The coordinate-pair values are referred to herein as (x^yi) 

2 for the source image, and (w/,v/) for the scanned image, for l</<6, 

3 which, for the first strip, correspond to (307) through (312) and 

4 (407) through (412) . The values of coordinate-pairs and their 

5 relationship to a line number of a scanned image line, the three 

6 tag-pairs associated with that scanned line, and the specified 

7 locations of the three lone vertical marks in the three 

8 synchronization-strips of the source images can best be shown by 

9 the following example. Considering the coordinate-pairs (310) , 

10 (311) and (312) and corresponding coordinate-pairs (410), (411) 

11 and (412), and, as before, using scanned line 250 as an example, 

12 the three corresponding coordinate-pair values would be: 

13 C*J>4) = (Fi,tf) = (70, 600.28), and (« 4 ,v 4 ) = K 5 I) = (39.17 5 250) 

14 (x5,ys)=(V 2 ,y* 2 ) =(5730, 598.85), and (us, v 5 ) = = (2392.50, 250) 

15 (x 69 y 6 ) = (V 3 y 3 )=(U390, 599.65), and (u 6 ,v 6 ) = (u* 39 L)= (4755.83, 250) 

16 

17 If (Ui,Vi) are the horizontal and vertical coordinate pair of pixels 

18 in the scanned image, and foo^) are the horizontal and vertical 

19 coordinate pair of pels in the source image, the two can be 

20 related by an affine transformation. In matrix equation form, the 

21 affine transformation is: 
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2 where the coefficient {U^Uy.UcV^V^Vc} are constants. Equation (1) 

3 can be expanded and rewritten in terms of four coordinate-pairs 

4 from the source image (307) , (308) , (310) , (311) and the four 

5 corresponding coordinate-pairs from the scanned image 

6 (407) , (408) , (410) , (411) as Equation (2). 
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8 Equation (2) is mathematically over-determined, needing only 

9 three corresponding coordinate-pairs instead of four, but 

10 Equation (2) as stated can be solved in a least-squares sense to 

11 give a first-instance of the coefficients {U x ,Uj,,U c ,V x ,V,,V c } , as 

12 shown in Equation (3) . 



13 



v v 



U\ 

vi 
u 2 
v 2 
u 4 
v 4 
us 

L V 5 



(3) 



YOR9-2003-0344 



-24- 



1 In the notation of Equation (3), the superscript T when applied 

2 to a matrix is taken to mean the transpose of that matrix, and 

3 the superscript -1 is taken to mean the inverse of that matrix. 

4 The positions of the scanned pixels contained in the 

5 quadrilateral having apexes (407) , (408) , (410) , (411) relative to 

6 the positions of the source image pels contained in the rectangle 

7 having apexes (307) , (308) , (310) , (311) are determined using 

8 Equation (1) , with application the first-instance of the 

9 coefficients {\J x ,\Jy,\J C9 Vxy y yc} . If the scanned image has fewer lines 

10 per inch and fewer pixels per inch in each line than the source 

11 image, as it does in the example embodiment, the affine transform 

12 of Equation (1) determines the two-dimensional coarsely aligned 

13 positions of the interpolated pixels, called herein points of 

14 interest. 

15 In the preferred embodiment, the interpolated pixel brightness 

16 values at the points of interest are determine by linear area 

17 interpolation in the two-dimensional w:v plane. A small square 

18 stub-area of the u:v plane [which is the plane of the scanned 

19 image] is isolated using the integer parts of the point of 

20 interest. For example, if the interpolated coordinates (w,v) are 

21 (1303.297,457.338), the point of interest is surrounded by the 

22 four pixel locations (1303,457}, (1304,457), (1303,458) and 
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1 (1304,458), and those four pixel locations, herein called 

2 vertices, define the square sub-area of interest. The fractional 

3 parts the coordinates {u,v} are called the residuals, (ur,vr), and 

4 are numerically (0.297,0.338). 

5 Linear interpolation in the sub-area begins by dividing the 

6 square into two triangular areas along a diagonal. The two 

7 triangular areas are herein called trihedrons. For example, of 

8 the two choices available for dividing the square, the upper-left 

9 to lower-right diagonal connecting vertices (1303,457) and 

10 (1304,458) is chosen. Then the trihedron in which the point of 

11 interest lies is determined. The criterion for selection is 

12 obvious: if ur>vr, the upper-right trihedron is chosen because it 

13 will completely enclose the point of interest; otherwise, if 

14 ur<vr, the lower- left trihedron is chosen. If the point of 

15 interest lies on the diagonal, ur=vr, either trihedron will do, 

16 since both will produce the same numeric result. 

17 Interpolation within the trihedron can be defined in terms of the 

18 natural coordinates of the point of interest. Natural coordinated 

19 have a very useful property in that they all lie in the domain 

20 (0,1) for all points within or on the boundary of the trihedron. 

21 In other words, all of the natural coordinates will be the in 

22 domain (0,1) for every interpolated point of interest, and at 

23 least one will not be in the domain for every extrapolated point 
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1 of interest. 

2 Interpolation within a trihedron can be defined in terms of the 

3 natural coordinates, a it of a point of interest as: 

4 fiuR,v R ) = a A f{uA,VA)+aBf(uB,VB) + acfiuc,vc), where Z at = 1 (4) 

5 The rectangular coordinates of the vertices are denoted as (u;,v/). 

6 The dependent variables at the vertices, which in the example are 

7 the pixel brightness values, are denoted as the functions ./(«/, v,). 

8 A further side condition is that the natural coordinates must sum 

9 to one. Expressing this relationship in matrix form: 



10 \fttiR 9 v R )] = [ J[u A ,v A ) fiu B ,v B ) y(wc,v c ) ] 



11 Based on the same natural coordinates, the coordinates of the 

12 point of interest can be written in terms of the rectangular 

13 coordinates of the vertices, as 



a A 
a B 
a B J 



(5) 



YOR9-2003-0344 



-27- 



UR 






UB 


uc 


a A 




VR 




VA 


VB 


vc 




(6) 


1 




1 


1 


1 







2 It is now possible to solve for the natural coordinates in terms 

3 of the rectangular coordinates of the vertices, as 
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5 Substituting for the natural coordinates, the interpolation 

6 equation can be written as 



\J(ur 9 vr)] = [ J(ua,va) Mb,v b ) fiu c ,v c ) ] 
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(8) 



8 For the example interpolation, the independent values ur and vr 

9 are 0.297 and 0.338, respectively. Because ur is less than vr, 

10 the lower-left trihedron having vertices (1,1), (0,1) and (0,0) 

11 is chosen. The natural coordinates are computed as: 
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1 All of the natural coordinates are positive and in the domain (0, 

2 1), and their sum is 1.0, as expected. 

3 It is worth noting in passing that since the sub-area of interest 

4 will always be a unit square having edges of unit length, the 

5 coordinates of its vertices will always be zeros or ones. Because 

6 of that, the inverse of the matrix of coordinates can have only 

7 values of 1, 0 , and -1 for its elements. Thus, the natural 

8 coordinates can always be computed without multiplication, using 

9 only addition and subtraction. 

10 Construction of a coarsely aligned replacement for a section of 

11 the scanned image can now be completed. The entire coarsely 

12 aligned replacement of the scanned image, including this first 

13 section and all other sections, will be referred to herein as the 

14 initial replacement image. For every pel location in or on the 

15 boundary of the rectangle defined by the four coordinate-pairs 

16 from the source image (307) , (308) , (310) , (311) , a point of 

17 interest is computed using Equation (1) applying the first 

18 instance of the coefficients {\J X9 Uy,Uc 9 Vx 9 V y9 Vc} - At each point of 

19 interest, a pixel brightness value is interpolated using Equation 

20 (8), also applying the first instance of the coefficients 

21 {U*,Uy,Uc,V x ,Vj,,V c } . The interpolated pixel brightness value is placed 

22 into an array of pixel brightness values that have a one-to-one 

23 positional correspondence with the source image pel location, used 
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1 to evaluate the point of interest. In this manner a section of 

2 the initial replacement image is constructed that has the same 

3 number of lines and pixels per line as the source image 

4 rectangle, and it is placed in the same location in the initial 

5 replacement image as the section bounded by the four 

6 coordinate-pairs (307) , (308) , (310) , (311) has in the source image. 

7 In the preferred embodiment, the interpolated pixel brightness 

8 values are binarized, that is, they are converted to one of two 

9 values and represented by a single binary bit. The criterion used 

10 for binarization in the preferred embodiment is simple 

11 thresholding; if the pixel brightness value is less than a 

12 threshold value it is set to 1; otherwise, it is set to 0 . An 

13 example threshold value is 50% of the maximum brightness value. 

14 Once binarized, the pixel is referred to as a pel. A binarized 

15 pixel value is a special case of a half toned pel value. Half toned 

16 pel values can be represented by more than one bit; binarized pel 

17 values are represented by a single bit. 

18 Construction of the remaining sections of the initial replacement 

19 image is continued using the next groups of four coordinate-pairs 

20 from the source image (308), (309), (311), (312) and the group of 

21 four corresponding coordinate-pairs from the scanned image 

22 (408), (409), (411), (412) to produce a second instance of the 

23 coefficients {U x ,Uj,,U c> V x ,V^,V c } . The section of the source image 

24 bounded by the four coordinate-pairs (308), (309) , (311), (312) is 
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1 constructed in like manner as hereinabove by applying the second 

2 instance of the coefficients {U X ,U^,U C ,V„V^,V C } . This completes 

3 construction of the first strip of the initial replacement image. 

4 A second strip is defined by six coordinate-pairs chosen numbered 

5 sequentially (310) through (315) from the source image and (410) 

6 through (415) from the corresponding scanned image. The process 

7 of constructing a second strip of the initial replacement image 

8 is completed in a like manner as for the first strip. Additional 

9 strips are constructed in like manner until all strips of the 

10 first web segment are completed. Then the process defined for the 

11 first web segment is repeated for every subsequent web segment 

12 until the entire document is completed, and then repeated as many 

13 times as necessary for all document copies that define a printing 

14 job. 

15 A flow chart of an example coarse image alignment process is 

16 shown in Figure 8. Referring to Figure 8, the digitized images of 

17 all pages to be printed are supplied in page sequential order 

18 (801). A process in the printer controller [Figure 1., (119)] 

19 groups the pages into logical image pairs. By logical image pairs 

20 it is meant, for example, that if the final document is to have 

21 100 pages and is to be bound so that pages 50 and 51, printed on 

22 the obverse side of the web, face each other, then page numbers 2 

23 and 99 form a left-right logical image pair and are the first 
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1 logical image pair directed to the obverse print engine, and page 

2 numbers 100 and 1 form a second left-right logical image pair and 

3 are the first logical image pair directed to the reverse print 

4 engine. The printer controller process also embeds three 

5 synchronization-strips into the left margin, right margin and 

6 center gutter of all logical image pairs, thus forming marked 

7 image pairs (803) . 

8 The first marked image pair is selected for each print engine and 

9 represent image pairs that will be printed back-to-back on the 

10 web (805) . The marked image pairs are printed (807) and scanned 

11 (809) in the Print Verification Unit [Figure la, (157) and (158)] 

12 forming an obverse scanned image pair and a reverse scanned image 

13 pair. The horizontal and vertical positions of features within 

14 the synchronization-strips are tracked using pixel values from 

15 the scanned images (811) . The purpose of tracking is to establish 

16 positional synchronization with the appropriate marked image 

17 pair. 

18 Process steps from (813) through (827) apply to the obverse print 

19 engine and the obverse side of the web; a procedurally identical 

20 process applies to the reverse print engine and the reverse side 

21 of the web. Therefore, process steps (813) through (827) are 

22 executed simultaneously and in parallel for the obverse and 

23 reverse sides of the web. The parallel processes merge at step 
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1 (829) 



2 Images in the scanned image pair and the corresponding marked 

3 image pair are separated into right and left images, and the left 

4 images are selected (813). The selected marked image and scanned 

5 image are divided into corresponding horizontal strips based on 

6 tracked vertical synchronization-strip positions, and the first 

7 horizontal strips, one from each image, are chosen (815) . 

8 An affine transforms is evaluated based on the tracked horizontal 

9 positions of the synchronization-strips in the top and bottom 

10 lines of the chosen horizontal strip (817) . The affine transform 

11 is, in turn, used to evaluate points of interest in the scanned 

12 image strip that correspond to each and every pel location in the 

13 marked image strip (819) , and a pixel value is interpolated at 

14 each point of interest in the scanned image (821) . 

15 Each interpolated pixel value is placed into an appropriate left 

16 or right initial replacement image pair at the coordinates of the 

17 marked image pel location that created the point of interest used 

18 for its interpolation (823). In this manner, the scanned image 

19 (left or right, whichever is being processed on the particular 

20 pass through the process) is increased in size horizontally and 

21 vertically, and the interpolated pixels form an initial 

22 replacement image equal in size to the marked image. Process flow 
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1 continues at this point with a fine alignment sub-process shown 

2 in Figures 9 which will be described subsequently, and then 

3 returns here. 

4 If all the strips of the chosen marked image have been processed 

5 (824), the process proceeds to (826); otherwise, the next 

6 horizontal strip in the sequence is chosen (825), and process 

7 steps (815) through (824) are repeated. 

8 Process flow continues at this point (826) with an image 

9 comparison sub-process shown in Figures 10 which will be 

10 described subsequently, and then returns here. 

11 If all the strips have been chosen and both the left and right 

12 sides of the marked image have been processed, the parallel 

13 obverse and reverse processes merge and proceed to (829) ; 

14 otherwise, the right image is selected (828) and process steps 

15 (807) through (827) are repeated. 

16 If all the marked image pairs are complete (829), the process 

17 terminates (831) with all source images printed, scanned, 

18 coarsely aligned, finely aligned and compared; otherwise, the 

19 next marked image pair in sequence is selected (830), and process 

20 steps (807) through (829) are repeated. 
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1 Compensating the scanned pixels for sensor variations 



2 Pixel brightness values, denoted as the functions fluuvi), are not 

3 taken directly from a scanned image of the second image stream. 

4 Since the value A^r^r) is computed by interpolation and is a 

5 weighted average of values of its three surrounding neighbors 

6 A u a,v a \ Amb,v b \ y(wc,vc) , and since in the preferred embodiment each 

7 pixel value is digitized by a separate sensor in one of two 

8 physically separate linear arrays [top or bottom of the web] in 

9 the Print Verification Unit (Figure 1,(113)), each of the sensing 

10 elements needs to be compensated individually in terms of the 

11 black and white values it senses. If there are J pixel brightness 

12 sensing elements for each line, and Bj and Wj are the measured 

13 pixel brightness values when the sensor j (izj^J) is sensing a 

14 printed black test patch and blank paper, then, for line i, the 

15 compensation equation is: 



17 where fiupVi) are the uncompensated brightness values measured by 

18 the individual sensing elements. 

19 Fine alignment of images of the first and second streams 

20 Further alignment of pels in the initial replacement image 



16 




(10) 
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1 relative to corresponding pels in the source image is usually 

2 needed in the horizontal direction [across the web] (and, 

3 fortunately, much less so in the vertical direction [along the 

4 web]) because of the unpredictably bad behavior of paper. In the 

5 small dimensions of a single pel, paper behaves much like the 

6 wood from which it is made. Paper warps, shrinks and expands 

7 based on its moisture content, and its moisture content varies 

8 rapidly based on the temperature and humidity of objects it comes 

9 in contact with. Further fine alignment of pels in the initial 

10 replacement image produces yet another image, herein called the 

11 final replacement image. 

12 Fine alignment of pels requires slicing the horizontal strips of 

13 the initial replacement image into vertical stripes. In the 

14 preferred embodiment, fine alignment will be facilitated by 

15 computing the one dimensional [horizontal] cross-correlation 

16 function of the pel values of the initial replacement image and 

17 the source image on a strip-by-strip and a stripe-by-stripe 

18 basis. (For convenience, the cross-correlation function is also 

19 referred to herein as the correlation function.) In the preferred 

20 embodiment, the stripes are approximately 0.25 inches wide and 

21 one inch high. At the example 600 pels per inch and 600 lines per 

22 inch, stripes are 600 lines high and 150 pels wide. In the 

23 preferred embodiment, as a convenience for the computer the width 

24 of the stripe is increased to 160 pels; with each pel value 



YOR9-2003-0344 



-36- 



1 represented by one bit, each line of stripe width is represented 

2 in the computer's memory by five 32-bit words. As a computational 

3 convenience also, each line of the initial replacement image is 

4 expanded by 64 pels of white (two 3 2 -bit words set to zero) on 

5 its left side. 

6 Evaluation of the horizontal cross -correlation between the 

7 initial replacement image and the source image is begun with the 

8 leftmost stripe of the strip. Binary pel values are referenced 

9 thirty- two at a time, that is, as one 32-bit word. The index 

10 m, (l<w<600) is used to reference the lines of the strip and the 

11 index n,(l<n<5) is used to reference groups of 3 2 -pel values 

12 [32-bit words] in each line of the stripe. 

13 The cross -correlation function is denoted as C(k), where k is the 

14 number of pel positions of horizontal offset in lines of the 

15 initial replacement image relative to pel positions in lines of 

16 the source image, and where k is positive if pels of the initial 

17 replacement image are offset to the right of pels in lines of the 

18 source image. Since the two images have one bit per pel, very 

19 significant simplification can be achieved in computation of the 

20 cross-correlation function. Multiplication of pixel brightness 

21 values in the cross -correlation function can be replaced by a 

22 bit-wise logical exclusive-or, symbolized herein as ®, which 

23 operates on thirty- two pixel values at a time. Figure 5 shows the 
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1 defining truth-table of the exclusive-or operator for single bit 

2 operands (505) . The summation of sub-products in the 

3 cross-correlation function is replaced by a sum of the 

4 count-of -ones remaining after application of the 32-bit 

5 exclusive-or operation. 

6 For an example single stripe, five words in width and with pels 

7 in lines of the initial replacement image offset thirty- two pel 

8 positions [one word] relative to pels in lines of the source 

9 image, the simplified cross-correlation function is given by: 

5 600 

10 C(32)= S 2 countls(S(m^)®R [0 3(w,«~l)) (11) 

n = 1 m - 1 

11 where S(m,n) refers to 3 2 -pel words of the source image line, 

12 R[0](/w,n-l) refers to 32 -pel words of the initial replacement image 

13 line, and the function countls returns the count of bits in its 

14 32-bit argument that have the value 1. [Note that the index value 

15 h-1 causes pels of the initial replacement image lines to be 

16 offset thirty-two pel positions to the right of pels in the 

17 source image lines, which, initially, is redundantly indicated by 

18 the zero subscript of R] . 

19 Computation of another value in the cross-correlation function of 

20 the stripe, for example, C(30), requires that all pel values in 
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1 lines of the initial replacement image strip are physically 

2 shifted two pel positions to the left. The pel values of the two 

3 vacated pel positions at the right end of each line are filled - 

4 with zeros. Recall that 64 pel positions with values set to zero 

5 were previously appended to the left end of lines of the initial 

6 replacement image, so until the cumulative shift of pel positions 

7 exceeds 64, only the added zero value pels will be discarded. For 

8 the instance C(30), Equation (11) can be restated as: 

5 600 

9 C(30)= I I countls(S(/7T,n)®R [2 ](/n,n-l)) (12) 

n = \ m = \ 

10 with R[2](m,«-1) referring to pel values in the initial replacement 

11 image strip that have been physically shifted two pel positions 

12 to the left, as signified by the subscript [2] of R[2](w,tt-1). 

13 The procedure as described is repeated until the pel values have 

14 been shifted left, for the example two pel positions at a time, 

15 until the accumulative shift of pel values reaches 64, and the 

16 corresponding value in the cross-correlation function of the 

17 stripe, C(-32), is evaluated by: 

5 600 

18 C(-32)= I I countls(S(/n,K)®R [64] (w,«-l)) (13) 

n-\ m = 1 
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1 The process, left shifting pels two pel positions at a time, 

2 leads to 65 evaluations of the cross-correlation function for the 

3 single stripe. The best alignment of the two image stripes occurs 

4 at the least value of the cross-correlation function. [Note that 

5 a perfect alignment of the two image stripes, black pels to black 

6 pels and white pels to white, produced by some shift value k will 

7 produce a C(k) having a value of zero.] By taking the offset pel 

8 location k* of the smallest value of at the cross-correlation 

9 function evaluation C(k*) , together referred to herein as a pair, 

10 and two additional pairs of values, one on either side of the 

11 minimum, a parabola can be fit through the three pairs of values. 

12 The pel position value at the inflection point of the parabola, 

13 £ m i n/ is a very good estimate of the best alignment between the 

14 initial replacement image and the source image for the particular 

15 stripe in the particular strip. 

16 A parabola having a horizontal directrix is given by: 



17 



QJ)=P2* 2 +Pl*+P0 



(14) 



18 and the slope along the parabola by: 



19 




(15) 



20 Fitting a parabola through the three pairs of values selected 

21 above is specified in matrix form as: 
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2 The coefficients of the fitted parabola can then be solved for 

3 as : 
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5 The pel position value at the inflection point of the parabola, 

6 that is, where the slope of the parabola is zero, is given by: 



, -Pi 

^min — 2?2 ' ( 18 ) 



8 The pel position value, k m \ ni is a very good estimate of the 

9 horizontal alignment between the initial replacement image and 

10 the source image for the stripe, but it is only an average value 

11 of the alignment across the stripe, and it does not include any 

12 vertical alignment correction, which could also be necessary 

13 under extreme conditions. 

14 Those familiar with the art will recognize that there are 

15 pathological cases where the horizontal cross-correlation 
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1 function applied to a stripe will produce an indefinite 

2 inflection point. An example that produces an obvious indefinite 

3 inflection point is a stripe intended to be blank paper. In that 

4 case, regardless of the number of pel positions a pel in lines of 

5 the initial replacement image is displaced relative to a pel in 

6 lines of the source image, the count-of -ones will always have a 

7 zero sum, and £ m in can not be evaluated. A blank-paper screen that 

8 will detect this particular case uses the logical or operator to 

9 combine all pel values (32 at a time, depending on the functional 

10 word length of the example computer employed) of every line of 

11 the source image stripe. If the logical or of all pel values in 

12 the stripe has a zero value, then the stripe is intended to be 

13 blank paper. 

14 In addition to the blank-paper screen is a screen that will 

15 detect indefinite inflection points. The preferred embodiment of 

16 a screen requires the minimum value of the cross-correlation 

17 function, C(k*) , to have two defined neighbors, C(k* + \) and C(£*-l), 

18 and further, requires the positive difference between C(k*) and at 

19 least one of its neighbors to be greater than a specified 

20 threshold value. If either of the screens gives evidence of an 

21 indefinite inflection point, the inflection point from the same 

22 stripe in the immediately preceding strip is used instead. This 

23 requires, for completeness, that as part of the initialization 

24 process a carefully constructed source image guaranteed to 
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1 produce no indefinite inflection points is printed. This 

2 initialization will insure that all stripes in subsequent strips 

3 will have an immediately preceding stripes possessing defined 

4 values for their inflection points, and if necessary, the 

5 inflection points will be passed on from strip to strip. 

6 Referring to Figure 6, values of £ m in in every stripe of an 

7 example eight inch strip are shown. The edge alignments (601) and 

8 (602) are determined by the affine transform used in the course 

9 alignment to produce the initial replacement image. The 

10 intermediate val UeS Ot #min , each shown as a separate point such 

11 as (603), are determined for every stripe in the strip by 

12 repeating the described procedure while indexing horizontally. 

13 through the pel locations of the lines of the two images in the 

14 strip. Values between the intermediate values are determined by 

15 straight line interpolation. 

16 Thus, for every source image pel position, a further correcting 

17 horizontal offset of pel positions in the initial replacement 

18 image is estimated, and that further correcting horizontal 

19 offset, herein called a fine alignment, is applied in addition to 

20 the affine transform together forming the final replacement image 

21 from the corresponding scanned image of the second stream. Hence, 

22 if the interpolated values of fine alignment, shown in Figure 6 

23 as a function of source image pel position x, are denoted as the 
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1 function H(x), then by a modification of Equation (1) : 
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(19) 



3 the interpolated values of the final replacement image pixel 

4 values can be determined. This is accomplished by reapplying 

5 Equation (8) using value (w,v) determined from Equation (19), with 

6 the appropriate instances of the coefficients {U^,U^,U C ,V^,V^,V C } . 

7 Note that values of (x 9 y) will be pairs of integers, but values of 

8 (w,v) will generally not be. 



9 Construction of a finely aligned replacement for a section of the 

10 scanned image can now be completed. For every pel location within 

11 or on the boundary of the rectangle defined by the four 

12 coordinate-pairs from the source image, (307) , (308) , (310) , (311) , 

13 a point of interest is computed using Equation (19), applying the 

14 same first instance of the coefficients {Ujr.U^Uc.Vx.V^Vc} as before. 

15 At each point of interest a pixel brightness value is 

16 interpolated using Equation (8) , also applying the same first 

17 instance of the coefficients {U^U^U^V^V^Vc} as before. The 

18 interpolated pixel brightness value is placed into an array of 

19 pixel brightness values that have a one-to-one positional 

20 correspondence with the source image pel location used to 

21 evaluate the point of interest. In this manner a section of a new 
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1 image, herein called the final replacement image, is constructed, 

2 and it is placed in the same location in the final replacement 

3 image as the section bounded by the, four coordinate-pairs 

4 (307) , (308) , (310) , (311) has in the source image. In the preferred 

5 embodiment, the interpolated pixel brightness values are 

6 binarized. The criterion used for binarization in the preferred 

7 embodiment is again simple thresholding; if the pixel brightness 

8 value is less than a threshold value it is set to 1; otherwise, 

9 it is set to 0. An example threshold value is 50% of the maximum 

10 brightness value. 

11 A flow chart of a fine alignment process is shown in Figure 9. 

12 Fine alignment is actually a sub-process of coarse alignment, 

13 with connection shown in Figure 8 (823), but is described in 

14 detail here. 

15 Referring to Figure 9, fine alignment is supplied a horizontal 

16 marked source image strip and a corresponding initial replacement 

17 image strip as a strip pair (901) . The horizontal strips are 

18 divided into a number of corresponding vertical stripes, each 

19 having the same dimensions, and arranged to be processed 

20 sequentially (905) . In the example above the stripes are 160 pels 

21 wide and 600 lines high. The first pair of stripes, one marked 

22 source image stripe and a corresponding initial replacement image 

23 stripe, are chosen to continue. 
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1 The horizontal cross-correlation of the two vertical stripes is 

2 evaluated with the two stripes aligned initially, to produce a 

3 first correlation value (907). Then all pels of the initial 

4 replacement image stripe are offset horizontally first left and 

5 then right relative to the pels of the marked source image 

6 stripe, and a correlation value is computed at each offset. 

7 Additional offseting of pels and additional computing of 

8 correlation values is continued until the an optimal correlation 

9 value has been straddled. In the preferred embodiment shown 

10 hereinbefore, the optimal correlation value, and the computed 

11 value nearest to it, will be minimum values. Thus at least three 

12 correlation values between the vertical stripes of the chosen 

13 stripe pair are produced (907) . 

14 The offset and correlation value nearest the optimal correlation 

15 value and the offsets and correlation values of its two nearest 

16 neighbors are selected (909) . The three offsets and their 

17 corresponding correlation values are used to find, by 

18 interpolation, the offset of the optimal correlation value, 

19 herein called the optimal offset of the stripe (911) . The optimal 

20 offset of the stripe is paired with the pel position of the 

21 center of the vertical stripe that produced it, and the pair, 

22 herein called an optimal-offset pair, is retained for subsequent 

23 use. If the optimal offset of all stripes have been evaluated 
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1 (913), the process continues to (915); otherwise, the next stripe 

2 pair in the sequence is selected (914) and process steps (907) 

3 through (913) are repeated using the next selected stripe pair. 

4 When all stripe pairs have been selected and all the 

5 optimal-offset pairs evaluated, the optimal-offset pairs are 

6 arranged in pel position order to form discrete values of a fine 

7 alignment function (915) . The independent values of the fine 

8 alignment function are the horizontal pel positions of the marked 

9 source image, and the dependent values are the fine alignment 

10 corrections to those pel positions that have been determined from 

11 the cross-correlation evaluations. The fine alignment values are 

12 a measures of average paper distortion within each stripe. An 

13 example fine alignment function is shown in Figure 6. 

14 Intermediate values of the fine alignment function are determined 

15 by piece-wise linear interpolation, illustrated by the straight 

16 interconnecting lines in Figure 6. 

17 The final step of fine alignment process is to produce a strip of 

18 the final replacement image by the same interpolation methods 

19 used to produce the strip of the initial replacement image, with 

20 one important exception; for the scanned image pixel value 

21 interpolation, the horizontal coordinate of the pel of the source 

22 image is increased by the piece-wise interpolated fine alignment 

23 value in all reevaluations of the points of interest (917) . 
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1 

2 Developing comparison masks from the final replacement image and 

3 the source image 

4 In the preferred embodiment, where every source image and every 

5 final replacement image are binarized, that is, having one-bit 

6 pel values, direct comparison of pel values can be done in an 

7 efficient and straightforward manner by employing 32 -bit logical 

8 operators, depending on the computer used. To do the comparison, 

9 two masks are formed from the pel values of each of the two 

10 images. Each mask is itself another binariazed image with size 

11 equal to that of the source image, and in the preferred 

12 embodiment, also with size equal to that of the final replacement 

13 image. The four masks, specific to each image, will be referred 

14 to as the source dilation-mask, the source erosion-mask, the 

15 replacement dilation-mask and the replacement-erosion mask. In 

16 the preferred embodiment , black pel values are set to one and 

17 white pel values to zero. 

18 The source dilation-mask and the replacement dilation-mask are 

19 formed by a process herein called dilation. Construction of the 

20 source dilation-mask will be described. Dilation is started by 

21 setting all pel values in the source dilation-mask mask to zeros, 

22 which represent white pels. The pel position of each black pel in 

23 the source image forms a pel-center at the same location in its 
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1 corresponding source dilation-mask, and a pel-array centered at 

2 the pel-center is filled with black pels in the mask. In a 

3 preferred embodiment, a pel-array with a size designation of 

4 * three" is represented by nine black pels arranged in a 3 pel by 

5 3 pel square. If the dilation size designation is an even integer 

6 P, the pel array is represented by P 2 black pels in a P by P 

1 array, and the pel-center of the array is arbitrarily chosen to 

8 be the upper- left pel closest to the center of the pel array. 

9 Dilation-masks formed in this manner produce text, lines and 

10 halftone patterns that appear to be made of thicker strokes than 

11 those of the original image, that is, they appear "dilated." 

12 Referring to Figure 7, a source dilation-mask (703) produced from 

13 a source image (701) by the described method is shown. The 

14 replacement dilation-mask is generated from the final replacement 

15 image in the same manner. 

16 In a like manner the source erosion-mask and the replacement 

17 erosion-mask are formed by a process herein called erosion. 

18 Construction of the source erosion-mask will be described. 

19 Erosion is started by setting all pel values in the source 

20 erosion-mask mask to ones, which represent black pels. The pel 

21 position of each white pel in the source image forms a pel-center 

22 at the same location in its corresponding source erosion-mask, 

23 and a pel-array centered at the pel-center is filled with white 

24 pels in the mask. In a preferred embodiment, a pel-array with a 
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1 size designation of * three" is represented by nine white pels 

2 arranged in a 3 pel by 3 pel square. If the erosion is an even 

3 integer P, the pel array is represented by P 2 white pels in a P 

4 by P array, and the pel -center of the array is arbitrarily chosen 

5 to be the upper-left pel closest to the center of the pel array. 

6 Erosion-masks formed in this manner produce text, lines and 

7 halftone patterns that appear made of thinner strokes than those 

8 of the original image, that is, they appear "eroded." Again 

9 referring to Figure 7, a source erosion-mask (705) produced from 

10 a source image (701) by the described method is shown. The 

11 replacement erosion-mask is generated from the final replacement 

12 image in the same manner. 

13 Detection of significant defects in the printed copy 

14 Comparison of the two images is done using the generated masks. 

15 Dilation and erosion masks are thus used to allow tolerance of a 

16 small positional uncertainty in the alignment of the final 

17 replacement image with the source image. In all cases, the source 

18 image is considered to be an errorless copy of what was intended 

19 to be printed. The final replacement image, whose origin is 

20 directly traceable to a scan of the printed copy, may not be 

21 error free because of defects in the printing process . Detection 

22 of these defects is the essence of the present invention. In the 

23 preferred embodiment, the source image, the source dilation-mask, 
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1 the source erosion-mask, the final replacement image, the 

2 replacement dilation-mask, and the replacement erosion-mask are 

3 all binarized images with each pel represented by a single binary 

4 bit; the binary pel values of the six images will be referred to 

5 as S(x,y), S D (x,j>), S e (x,>0, T(x,j/), T D (x,y), and T E (x,>>)/ respectively, and 

6 where (x,y)are the indices of the image line, y, and pel location 

7 within that line, x. Comparison of the pel values of specific 

8 pairs of these images can be done using logical operators. 

9 The preferred embodiment for detecting unintended application of 
10 ink in a single pel on the printed page is the simple equation: 



11 (20) 

12 where the logical operators ® [or] and ® [exclusive-or] are 

13 defined in Figure 5 (501) and (503) . If Equation (20) is 

14 evaluated to be zero, no unintended ink is detected. Those 

15 familiar with the art will recognize that alternatives. 

16 combinations can be used in place of Equation (20) with similar 

17 results; for example: 

1 8 (Sd(x,^) © T(jc,>;)) ® S D (x,y) = 1 

1 9 (S(x y y) 0 T E (x,y)) ® S(x,y) =1 (21) 
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1 Since digital computers are capable performing bit-wise logical 

2 operations on 32 pels with a single instruction, depending on the 

3 computer used, significant efficiency of comparison of pel values 

4 is obtained. 

5 The preferred embodiment for detecting the unexpected absence of 

6 ink in a single pel on the printed page is the equation: 

7 (SeCc, jO <8> 7 D (x,y)) ® S E (x, j>) = 1 (22) 

8 where the logical operator <g> [and] is defined in Figure 5 (502) . 

9 If Equation (22) is evaluated to be zero, no unexpected absence 

10 of ink is detected. Those familiar with the art will also 

11 recognize that alternatives combinations can be used in place of 

12 Equation (22) with similar results. 

13 To obtain a lower false detection probability, a small cluster of 

14 detected pels, for example, a 2 pel by 2 pel square area, can be 

15 used in which a detection must be made in two adjacent pels at 

16 the same location in two adjacent lines before a detection is 

17 deemed to be significant. This is consistent with the premise 

18 that at 600 pels per inch and 600 lines per inch no meaningful 

19 information is conveyed to a human viewer by a single pel. 

20 A flow diagram of an image comparison sub-process is shown in 
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1 Figure 10. Image comparison is actually a sub-process of coarse 

2 alignment, with connection shown in Figure 8 (826) , but is 

3 described in detail here. 

4 The sub-process is supplied a marked source image and a 

5 corresponding final replacement image (1001) . Note that the 

6 supplied images are full page images, not image strips, and by 

7 process design they have equal horizontal and vertical pel 

8 dimensions. If the pel values of either image are not one-bit 

9 values, they are converted to one-bit values by thresholding 

10 (1003) such that black pel values are 1 and white pel values are 

11 0. 

12 Dilation and erosion masks are formed for each image. Those for 

13 the marked source image are referred to as So and Se (1005) , and 

14 those for the final replacement image are referred to as Td and 

15 T E (1007). The masks have horizontal and vertical pel dimensions 

16 equal to those of the marked source image. The pel positions in 

17 the masks are ordered sequentially by positions in a line, left 

18 to right, then by lines from the top of the image to the bottom. 

19 A first pel position is selected from the sequence (1009) . 

20 Unintended application of ink in the selected pel is detected by 

21 forming a first intermediate result as the logical or of selected 

22 pel value of Sd with the selected pel value of T E , then the 
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1 logical exclusive-or of that first intermediate result with the 

2 selected pel value of S D (1011) . If the final value of the two 

3 operations is a 1, the unintended application of ink in that pel 

4 is detected, and the pel location of the detection is noted for 

5 later use. 

6 The absence of intended ink in the selected pel is detected by 

7 forming a second intermediate result as the logical and of 

8 selected pel value of S E with the selected pel value of Td, then 

9 the logical exclusive-or of that second intermediate result with 

10 the selected pel value of S E (1013). If the final value of the 

11 two operations is a 1, the absence of intended ink in that pel is 

12 detected, and the pel location of the detection is noted for 

13 later use. 

14 A defective printed copy is declared (1017) if at least one of 

15 the following conditions exists: 

16 a) Two or more adjacent pels in the same pel positions in two 

17 or more adjacent lines have unintended ink application. 

18 b) Two or more adjacent pels- in the same pel positions in two 

19 or more adjacent lines have the absence of intended ink. 

20 If all image pels have been chosen, the sub-process returns 
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1 control (1019) ; otherwise, the next pel position in sequence is 

2 selected (1020) and steps (1011) through (1019) are repeated. 

3 Streak and splotch detection 

4 For very high quality printing, detection of streaks and 

5 splotches can be used to reject printed copies that, although 

6 still readable by a human viewer, are cosmetically undesirable. 

7 Minor variations of the methods documented hereinbefore can be 

8 used. For these purposes, a second criterion for binarization of 

9 the final replacement image is used. Although the criterion is 

10 still simple thresholding, a different threshold value, for 

11 example 25% instead of 50%, is used. This lower threshold value 

12 causes relatively light shades of gray to be binarized into 

13 black. Additional screening of the detection of unexpected black 

14 must be used to avoid excessive detection rates caused by 

15 insignificant flaws. For streaks, an additional screen requires a 

16 detection to occur in the same pel location in a large number of 

17 consecutive lines, for example, 100 lines, before the combined 

18 detection is declared a streak. For splotches, another additional 

19 screen requires a detection to occur in a relatively large area, 

20 for example 90% of pels in a 100 pel by 100 pel area, before the 

21 combined area detection is considered a splotch. 

22 Although a number of variations of the preferred embodiments have 
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1 been noted, it will be clear to those skilled in the art that 

2 many other variations and modifications to the disclosed 

3 embodiments can be effected without departing from the spirit and 

4 scope of the invention. An obvious variation would be the use of 

5 a computer capable of processing 64-bit words instead of 32-bit 

6 words. The described embodiments ought to be construed to be 

7 merely illustrative of some of the more prominent features and 

8 applications of the invention. Other beneficial results can be 

9 realized by applying the disclosed invention in a different 

10 manner or modifying the invention in ways known to those familiar 

11 with the art. 

12 Although the preferred embodiment of the present invention is for 

13 monochrome digitized images and monochrome printed copy, those 

14 skilled in the art will understand that for a color image, each 

15 pixel of the digitized image has associated values representing 

16 the magnitudes of average brightness of its at least three color 

17 components represented in three or more color planes. The color 

18 components are associated with spectrally dispersed primary 

19 colors used to represent a broad range of colors in the visible 

20 color spectrum, and the values of the at least three color 

21 components are the relative brightness of the three primary 

22 colors used to represent a particular color. 

23 The preferred embodiment of the present invention will use only 
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1 monochrome source images having pel values of one binary bit. The 

2 source images could include any content but in the preferred 

3 embodiment are composed primarily of text and simple geometric 

4 structures, such as outlining boxes or grids. Specifically, the 

5 source images of the preferred embodiment do not contain subparts 

6 that are half toned images of natural or artificial scenes. 
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